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All programs have requirements. For these requirements to be met, there must be 
a means of measurement. A Technical Performance Measure (TPM) is defined to 
produce a measured quantity that can be compared to the requirement. In practice, 
the TPM is often expressed as a maximum or minimum and a goal. Example TPMs 
for a rocket program are: vacuum or sea level specific impulse (Isp), weight, 
reliability (often expressed as a failure rate), schedule, operability (turn-around time), 
design and development cost, production cost, and operating cost. Program status is 
evaluated by comparing the TPMs against specified values of the requirements. 
During the program many design decisions are made and most of them affect some or 
all of the TPMs. Often, the same design decision changes some TPMs favorably while 
affecting other TPMs unfavorably. The problem then becomes how to compare the 
effects of a design decision on different TPMs. How much failure rate is one second of 
specific impulse worth? How many days of schedule is one pound of weight worth? In 
other words, how to compare dissimilar quantities in order to trade and manage the 
TPMs to meet all requirements. One method that has been used successfully and has a 
mathematical basis is Utility Analysis. 

Utility Analysis enables quantitative comparison among dissimilar attributes. It 
uses a mathematical model that maps decision maker preferences over the tradeable 
range of each attribute. It is capable of modeling both independent and dependent 
attributes. Utility Analysis is well supported in the literature on Decision Theory. It 
has been used at Pratt & Whitney Rocketdyne for internal programs and for 
contracted work such as the J-2X rocket engine program. 

This paper describes the construction of TPMs and describes Utility Analysis. It 
then discusses the use of TPMs in design trades and to manage margin during a 
program using Utility Analysis. 


Nomenclature 


ATP 

= Authority to Proceed 

IOC 

= Initial Operating Capability 

IR&D 

= Internal Research and Development 

NASA 

= National Aeronautics and Space Administration 

O/S 

= Operations and Support 

PWR 

= Pratt & Whitney Rocketdyne 

RS 

= Rocket System 

T 

= Time 

TPM 

= Technical Performance Measure 
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Introduction 

A LL programs have requirements. For these requirements to be met, there must be a means of 
measurement. A Technical Performance Measure - a TPM - is defined to produce a measured 
quantity that can be compared to the requirement. In practice, the TPM is often expressed as a maximum or 
minimum and a goal. Examples of TPMs for a rocket program are: vacuum or sea level specific impulse 
(Isp), weight, reliability (often expressed as a failure rate), schedule, operability (turn-around time), design 
and development cost, production cost, and operating cost. Program status is evaluated by comparing the 
TPMs against specified values of the requirements. During the program many design decisions are made 
and most of them affect some or all of the TPMs. Often, the same design decision changes some TPMs 
favorably while affecting other TPMs unfavorably. The problem then becomes how to compare the effects 
of a design decision on different TPMs. How much failure rate is one second of specific impulse worth? 
How many days of schedule is one pound of weight worth? In other words, how to compare dissimilar 
quantities in order to trade and manage the TPMs to meet all requirements. 

One Possible Approach - Utility Analysis 

One method that has been used successfully at Pratt & Whitney Rocketdyne (PWR) is Utility Analysis. 
Utility Analysis enables quantitative comparisons among dissimilar attributes. In other words, it takes 
apples and oranges and converts them all into peaches. For programs that have competing objectives, a 
multi-attribute (multi-TPM) utility function can be used to resolve the tradeoffs among the different TPMs. 

Utility is a mathematical representation of how much a decision maker values different amounts of one 
or more attributes. The mathematical model maps decision maker preferences over a trade space. A utility 
function, Ui(x,), is constructed for each attribute, modeling the strength of preference over the feasible range 
of the attribute. The utility functions are developed through a set of exercises with the decision maker 
evaluating hypothetical trades and degrees of risk aversion. Then the overall utility is (usually) a weighted 
sum of the values of the individual attribute utility functions. 

The focus on the decision maker directly captures the sensitivity of the decision maker’s expectations, 
thus the process captures the decision maker’s subjective influences. 

Utility Analysis can be used for non-linear and for dependent preferences. It is also very efficient for 
performing large numbers of trades. It requires some up-front effort to develop the model, but there is very 
little recurring effort needed during the program. The lack of recurring effort is in contrast to tools such as 
AHP or Accord which develop a new model for each individual trade. 

Utility Analysis is well-supported in academic literature on Decision Theory, and decisions based on 
utility satisfy “Axioms of Rational Behavior”. 1-5 

Overall, Utility Analysis provides a formal, traceable, repeatable, and consistent basis for making 
decisions within a program. 


Using Utility Analysis 

The first step in using Utility Analysis is to determine what TPMs are going to be used to manage the 
trades or the program. For a rocket engine, a common set of TPMs is: performance, weight, reliability, 
operations, cost, and schedule. Others could be used, but this set is typical. Each of these TPMs needs to be 
broken down to one or more measurable attributes. Figure 1 shows a set that could be used. For example: 
cost is broken down into non-recurring cost, recurring cost, and operations and support cost. For each of the 
eight attributes shown in the figure a utility curve is then constructed. Then the overall utility will be a 
function of the eight attributes that together represent the overall strength of the customer preference: 

U (jtj , x 2 ,...x n ) = k l u 1 (Xj ) + k 2 u 2 (x 2 ) + ... + k n u n (x n ) 
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Figure 1. Typical set of attributes for a set of TPMs. 


Generating the Individual Utility Curves 

The goal is to determine the customer preferences, so the involvement of the decision maker is 
imperative. The decision maker may defer to judgments of a number of individuals with specific domain 
knowledge, but in that case, the number should be kept small because the time to perform the process 
described below increases quickly as the number of people involved increases. 

The process is: considering one attribute at a time, the decision maker is asked to trade outcomes that 
are certain against outcomes that are uncertain. This will establish risk aversion versus risk seeking 
attitudes over the attribute range. It identifies “pain thresholds” and points of diminishing return. 

For each attribute, a minimum and a maximum value that bounds the trade space is selected. These 
should be extreme enough to serve as a boundary, but not so extreme as to be unrealistic. The maximum 
value of the attribute is assigned a utility of 1 and the minimum value is assigned a utility of 0. For all of 
the points between these two, the utility is scaled such that the expected utility of a lottery of outcomes is 
equal to the utility of the certain outcome that the decision maker feels is of equal preference. 

In more process oriented terms, the utility curve can be generated by using the following process. 


Step 1. Agree on the minimum and maximum possible values of the attribute in question. This agreed- 
upon minimum value is referred to in the questions that follow as x 0 and the maximum value is 
referred to as x/. 

Step 2. Arbitrarily select a value between x 0 and Xj. This value is referred to below as x*. x* should be 
close enough to either x 0 or xj that the respondent should have a clear preference in answering 
the following question: 


Question 1: “You have two alternatives, one with an outcome that is certain, and one with an outcome 

that is uncertain. If you select the first alternative, you are guaranteed an outcome of x . If 
you select the second alternative, there is a 50% chance the outcome will be x 7 , and a 50% 
chance the outcome will be x (h Which of the two alternatives would you prefer?” 

Step 3. Select a new x*, this time choosing a value toward the opposite end of the spectrum, so that the 
respondent, in answering Question 1, would have an equally clear preference for the opposite 
alternative. Repeat Question 1 with the new value forx*. 

Step 4. Continue to repeat Question 1, each time selecting a new value of x* from alternating sides of the 
X spectrum, such that each time the respondent’s preference will become less clear and the 
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decision becomes more difficult. When a value of x* is reached for which the decision cannot be 
made and there is no clear preference between the alternatives of Question 1 , we’ll say we have 
reached an indifference point and the value of x will be denoted by x 0 .s. 

Step 5 . Repeat Steps 2 - 4 , replacing x 0 with x 0 .s. The resulting indifference point is denoted by x 0 75. 

Step 6. Repeat Steps 2-4, replacing xj with x 05 . The resulting indifference point is denoted by x 0 25 - 

Following this procedure will result in the determination of three interior points on the utility curve: x 0 .5, 
X0.75, and Xq 25' As a sanity check, these three points may correspond roughly to the following quantities 
from the decision maker’s perspective: 

x 0 .5 just enough (of this attribute) to keep the program viable; 

X0.75 point with enough margin that it is a “slam-dunk”; 

X0.25 probably untenable, but enough to “prolong the pain”. 

The complete utility curve for this particular attribute is the best fit through the five points x 0 , X0.25, x 0 .5, 
X0.75, and Xj. An example of the type of curve to be expected is shown in Figure 2 . 



Utility Curve Characteristics 

If assessed over a broad enough range, most utility curves will typically take an “S” shape with three 
distinct regions as illustrated in Figure 3 . 



Figure 3. Interpretation of typical utility curve. 
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Examining Figure 3, a number of observations can be made. Portions of the curve that are concave 
down indicate regions where the decision maker is risk averse. Portions that are concave up indicate 
regions where the decision maker is risk seeking. And linear segments of the curve indicate regions in 
which the decision maker is risk neutral. The slope of the utility curve indicates the relative worth of 
marginal improvement about any point on the curve. 

Risk aversion means that the benefit of improving attribute x by an amount 8 is less than the harm in 
changing it by -8. On the risk averse portion of the curve the decision maker would rather not take a 50-50 
chance of changing the attribute value by +/-8. Risk seeking has a similar, but opposite, interpretation. 

The risk averse portion of the curve approaching the utility of 1 is a reflection of the law of diminishing 
returns, while risk seeking is observed as the utility of 0 is approached. 

Some decision makers claim to be risk neutral in all regions above some minimum threshold - in effect 
denying the law of diminishing returns. Whether this is a reasonable attitude is a subject of some debate 
and should be discussed carefully with the decision maker and treated accordingly in the construction of the 
curve. 

The concepts of risk aversion, risk seeking, and risk neutrality are very useful in interpreting the shape 
of the utility curve and can be used to “troubleshoot” curves that do not come out looking the way the 
decision maker expected. 

Depending on the range of the attribute chosen, the utility curve can be any subsegment of the general 
shape shown in Figure 3. 


Scaling Factor Determination 

Each utility curve in the overall utility function has a scaling (or weighting) factor. The utility curve is 
of the form Ui = kjU^Xj). The scaling factor, k l? for the utility curve for each attribute is set by pairwise 
comparison with other attributes. Using Figure 4, Xj + represents the value of Xi with the greatest utility. If 
the decision maker prefers point A to point B, then > k 2 . Then finding point A’, which has equal value to 
the decision maker as point B, sets the value of k[ relative to k 2 . Scaling n utility curves requires n-1 
pairwise comparisons of attributes for n equations in n unknowns. The sum of the weighting factors equals 
1, i.e., Skj = 1. As an example, assuming a case with four attributes, then there are seven possible equations 
available plus the fact that the sum of the k’s equals one: 


k t = k 2 u(xO 
ki = k 3 u(xi’) 
k 1 = k4u(x 1 ’) 
k 2 = k 2 u(x 2 ’) 


k 2 = k 3 u(x 2 ’) 
k 2 = k4u(x 2 ’) 
k 3 = k4u(x 3 ’) 
Skj = 1 • 



Figure 4. Setting the utility curve weighting factors. 
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Only four equations are needed to determine the k’s. Pick the combinations that have attributes that are 
reasonably comparable, e.g., schedule and cost, not cost and center of gravity. Then since the problem is 
over determined, choose a couple of others as cross-checks. 

Dual-Attribute Utility Curve Construction 

What happens if the decision maker’s preferences are not independent? In other words, what if the 
utility of Xi depends on the value of x 2 , and vice versa? Assume that the utility of x 2 given a particular value 
of Xi, u(x 2 |xi), can be described by a family of functions over x 2 whose parameters are a function of X\. The 
entire dual-attribute utility surface can be generated by calculating three single-attribute utility curves. 

Figure 5 shows an example of a dual attribute utility where the weight (wt), denoted as x 2 , and the 
center of gravity (eg), denoted as Xj, are preferentially dependent. Looking at Figure 5, the upper and lower 
utility curves are calculated using the procedure described earlier in the section “Generating the Individual 
Utility Curves,” with one difference. In this case, the utility at points f and g of Figure 5 are known (1 and 
0). However, the utility at points e and h are not known. It is necessary to independently assess the utilities 
of points e and h relative to point f. This is accomplished by again asking the decision maker to choose 
between an outcome that is certain and an outcome that is uncertain. In this case, the certain outcome is the 
one represented by point e (or h), and the uncertain outcome is one that results in point f with probability p, 
and point g with probability 1-/7. The probability p e (or p h ) which leads to the decision maker being 
indifferent to the uncertain outcome relative to the certain outcome is then, by definition, equal to the 
expected utility of point e (or h).. 

Having developed two consistently scaled conditional utility curves for x 1? a conditional utility curve for 
x 2 is then calculated using the same procedure, and then scaled such that its endpoints meet the boundary 
conditions established by the two Xj curves. 



Figure 5. Conditional single-attribute utility curves. 

Now assume that the same functional form can be used to generate a family of conditional utility curves for 
x2, given different values of xl. For the following discussion, an arctangent form will be assumed. Then 
the functional form is: 

U(x 2 |xi) = Cj atan[C 2 (x 2 + A)] + B 

Where Ci and C 2 are constants and A and B are functions of Xj. A series of parallel slices are taken at 
various values of Xi that span its range. Nine slices work well, but the number must be such that curve fits 
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of A and B can be performed. For each slice the boundary conditions (the two Xj curves) are known, 
yielding two equations in two unknowns: 

Ui(xi,x 2 ‘) = Q atan[C 2 (x 2 ‘ + A^))] + B(x,) 


and 


ui(xi,x 2 °) = C, atan[C 2 (x 2 ° + A(x,))] + B(x,) 


where 

x 2 * is the x 2 for best utility and x 2 ° is the x 2 for worst utility. 

This process generates a set of points for A(xj) and B(xj). These sets of points are then curve fit to yield the 
functional forms of A and B. Figure 6 shows the surface build up. 



Figure 6. Utility surface construction. 

Figure 7 shows the resultant utility surface for this example. If another set of parallel slices are taken at 
values of constant utility, Figure 8 results. Figure 8 is an example of a set of indifference curves. For any 
given utility plotted on the figure, there is no change in utility (preference) between weight and center of 
gravity along that line. 



Figure 7. Dual-attribute utility function Ui 2 (x 1 ,x 2 ). 
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Figure 8. Example of indifference curves. 

Figure 9, another indifference curve, illustrates that the exchange factor between any two attributes, 
whether or not preferentially dependent, can, and probably will, be different depending on the current value 
of each attribute. Consequently, during the course of a program, the optimum trades between them will 
vary. On any constant utility curve the direction of maximum increase of utility is perpendicular to the line 
of constant utility, and at different points in the program, this will change. To illustrate, consider the lower 
right quadrant of Figure 9. If the IOC date is around June 2011 and delivered payload is in the low 
50,000s, then increasing payload capability is more important to the decision maker than improving IOC 
date. Conversely, if the expected IOC date has slipped to January 2012, but payload has increased to the 
upper 50,000s, then shortening the schedule becomes more important to the decision maker than adding 
more payload. The indifference curves can be useful for quick visual information about the relative 
importance of various attributes. 



Figure 9. Direction of optimality can change as design matures. 
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Using Utility Functions for Trades 

For each option in the trade space, the values of the affected attributes are calculated. Then the utility 
function is evaluated for each option. The trade can then be compared on the basis of utility to the decision 
maker. Figure 10 shows the steps in this methodology. Note that the utility curves do not need to be 
regenerated for additional trades even with entirely new options. 


Traded Options 


Reference 

Configuration 


Viable Option 
1 


Viable Option 
2 


Viable Option 
n 


Option Assessment Translate Assessment Determine Overall 
Relative to Attributes into Utility Rankings 



Figure 10. Utility-based trade methodology. 

Figure 1 1 shows an example of a trade in a totally hypothetical program developing a vehicle 
architecture. The engine parameter values do not represent any real engines. The trade is between vehicles 
using seven and eight engines. The eight engine vehicle has better specific impulse, better non-recurring 
cost, and equal operation and support costs, whereas the seven engine vehicle is somewhat better in the 
other TPMs. Which configuration should be picked and how should the decision be justified? This could be 
a difficult decision. But by using Utility Analysis and applying it to this case, the results show that, for the 
preferences and degree of risk aversion exhibited by the decision makers for this example, there is a 
significant preference for the seven engine configuration. 


Performance 

7 Engines 

8 Engines 

Mission Avg. I SP 

435.8 sec 

436.4 sec 

Weight 

Total Vehicle Eng. Wt. 

48,547 lb 

49,732 lb 

Reliability 

Catastrophic failure prob. 

0.99984 

0.99972 

Ops 

Pr[T > 20 hrs] 

0.45 

0.50 

Cost 

Non-Recurring 

$859M 

$849M 

Recurring 

$632M 

$709M 

O/S 

$31. 1M 

$31. 1M 

Schedule 

ATP to 1st set delivery 

46.5 mo 

48 mo 

Overall Utility 

0.846 

0.753 


Figure 11. Example RLV system-level trade. 
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Utility Analysis provides a convenient method for developing trade factors for use during design. At 
any point in the program the values of the TPM attributes are known. In such a case, the partials for each 
attribute can be calculated. Figure 12 shows such a set for the previous example program. Now the impact 
of a change in any one attribute can be seen and comparisons, such as how much one second of specific 
impulse is worth in pounds of weight, can be calculated. For this example 0.1 seconds is worth 69.5 
pounds. 


^SP 

au/ax 1 

= 

0.025 / sec 

Weight 

d u/dx 2 

= 

-0.036 / klb 

Non-Recurring Cost 

di)/dx 3 

= 

-0.042 / $B 

Recurring Cost 

d U/dx 4 

= 

-0.028 / $B 

Reliability 

eu/dx 5 

- 

-0.00192 / unit prob. 

Schedule 

dl)/dx 6 

= 

-0.046 / mo 

O/S Cost 

dll/dx 7 

= 

-0.673 /$B 

Ops Turnaround 

dU/dx 8 

— 

-0.093 / unit prob. 


Figure 12. Use of utility surface factors for sensitivity analysis. 


Summary/Conclusions 

PWR has successfully used Utility Analysis on the RS-83, RS-84, and J-2X programs. It has been used 
on a number of NASA study contracts as well as on IR&D efforts. 

Utility Analysis is well-supported in the academic literature on Decision Theory. And decisions based 
on utility satisfy “Axioms of Rational Behavior”. Overall, Utility Analysis provides a formal, traceable, 
repeatable, and consistent basis for making decisions within a program. 
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